Systems and methods for creating a story board with forensic video analysis on a video repository

ABSTRACT

Systems and methods for creating a story board with forensic video analysis on a video repository are provided. Some methods can include storing a plurality of video data streams in a data repository, storing asynchronous streams of metadata of each of the plurality of video data streams in the data repository, identifying a first object captured by at least one of the plurality of video data streams, using the asynchronous streams of metadata to identify correlations or interactions between the first object and a plurality of other objects over time, and replicating a story of the first object.

FIELD

The present invention relates generally to forensic video analysis. Moreparticularly, the present invention relates to systems and methods forcreating a story board with forensic video analysis on a videorepository.

BACKGROUND

Forensic video analysis (FVA) is an investigative, post-event forensicscience, and the International Association for Identification (IAI) hasformally recognized FVA as a sub-specialty within the scientificdiscipline of forensic imaging. Specifically, EVA is the scientificexamination, comparison, and evaluation of video in legal matters. Thatis, FVA is the application of image science and domain expertise tointerpret the content of an image or the image itself in legal matters.Disciplines of FVA with law enforcement applications includephotogrammetry, photographic comparison, content analysis, and imageauthentication. For example, a forensic analyst may want to identifyinformation regarding the interaction of people and objects in an easyand accurate manner and may want a detailed incident management reportwith artifacts supporting the same for producing in a court of law.Similarly, a legal person may want to view sufficient and untamperedartifacts to articulate an incident in detail, including the people andobjects involved in the incident.

The Scientific Working Group on Imaging Technology (SWGIT) setsstandards for FVA and identifies the following tasks for the process ofFVA: technical preparation, examination, and interpretation. During theinterpretation process, specific subject matter expertise is applied todraw conclusions about video recordings or the content of thoserecordings. For example, drawing a conclusion about a video recordingcan include authenticating the video recording. Drawing a conclusionabout the content of a video recording can include comparing objects ordetermining that an object appears different in the video than theobject appears under normal lighting conditions due to the properties ofthe recording process, such as an infrared (IR) negative image effect onnatural fibers.

Any incident management report that the interpretation process generatesmust comply with the SWGIT standards, meet the requirements of ananalyst's agency, address a requestor's needs, and provide all relevantinformation in a clear and concise manner. However, there are currentlyno known systems or methods to perform FVA on a video repository of rawvideo data, as per the SWGIT standards, for example, to back track aperson or object to create a story board of various incidents involvingthat person or object or an associated person or object. Furthermore,there are currently no known systems or methods to perform aninvestigation on multiple associated persons, including tracking objectsassociated with such persons and interactions between such persons andobjects, or to create a story board of such persons and objects. This isbecause known systems and methods to interpret video and to generateincident management reports are manual and align with video data, notmetadata.

Notwithstanding the above, known video systems generate thousands ofvideo data streams per day, and one or more of those video data streamsmay contain representations of people or objects relevant to suspiciousactivities. However, most such video data streams exist only as datauntil they are overridden or flushed, not translated into metadata thatcan be a valuable data node for future FVA.

In view of the above, there is a continuing, ongoing need for improvedsystems and methods.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system in accordance with disclosedembodiments;

FIG. 2A is a first portion of an exemplary incident management reportthat can be generated in accordance with disclosed embodiments;

FIG. 2B is a second portion of an exemplary incident management reportthat can generated in accordance with disclosed embodiments;

FIG. 3 is an exemplary storyboard that can be generated in accordancewith disclosed embodiments; and

FIG. 4 is a flow diagram of a method in accordance with disclosedembodiments.

DETAILED DESCRIPTION

While this invention is susceptible of an embodiment in many differentforms, there are shown in the drawings and will be described herein indetail specific embodiments thereof with the understanding that thepresent disclosure is to be considered as an exemplification of theprinciples of the invention. It is not intended to limit the inventionto the specific illustrated embodiments.

Embodiments disclosed herein can include systems and methods forcreating a story board with FVA on a video repository. It is to beunderstood that systems and methods disclosed herein can execute FVAoffline. That is, in some embodiments, the FVA as disclosed anddescribed herein can be executed on a stream of metadata, for example,when an original video data stream is unavailable.

In accordance with disclosed embodiments, video data streams or videoclips that are generated by video surveillance cameras and/or stored onvideo management servers or storage servers in video management systemscan be encapsulated with a proprietary data structure, including aproprietary file header. Accordingly, systems and methods disclosedherein can decapsulate the proprietary file header from a video datastream and execute a rapid analysis over the video data of the videodata stream to convert the video data into a stream of asynchronousmetadata. Then, the metadata can be stored in an atomic state in aforensic metadata repository, and information can be autonomouslyextracted from the metadata and converted into an incident managementreport.

As explained above, the process of interpretation in FVA includesapplying specific subject matter expertise to draw conclusions aboutvideo data streams or video clips or the content of those video datastreams or video clips. Such conclusions can include identifying variouscorrelations and interactions, for example, between persons and/orobjects depicted in the video data streams or the video clips over time.In some embodiments, systems and methods disclosed herein canautonomously identify such correlations from stored metadata toreplicate the story of an incident or the workflow of an event, toidentify interactions at locations of interest or information aboutareas of interest, to identify the time and the state of conditionsduring incidents, or to track persons or objects in metadata.

FIG. 1 is a block diagram of a system 100 in accordance with disclosedembodiments. As seen in FIG. 1, the system 100 can include a userinterface 110, a forensic management module 120, a forensic analysismodule 130, a third party interface 140, and a data repository 150.

The user interface 110 can include an incident management tool interfaceand an intelligent video management system (IVMS) client interface. Insome embodiments, the incident management tool interface can act as aplug in to existing architecture and can receive and transmit additionalparameters to the existing architecture that can be used to fabricateincident management reports. For example, the incident management toolinterface can act as a centralized interface for calibrating theadditional parameters, and the supplied additional parameters can causethe existing architecture to embed metadata associated with an incidentin an incident management report, including embedding metadata in videodata in an incident management report. In some embodiments, the IVMSclient interface can receive and transmit instructions to configure theforensic analysis module 130, to manage incidents, and to triggerstoryboard reporting.

The forensic management module 120 can include components and adaptersfor pulling video data and audio data from the data repository 150, fordecoding and decompressing raw data, for managing a metadata schema, fornormalizing metadata in a metadata database, for providing a storyboardinterface, for managing objects in the user interface 110, and formanaging plug ins for the forensic analysis module 130.

The forensic analysis module 130 can act as an analytical plug in engineto existing architecture and can classify persons and objects, detect,identify, and track persons and objects, and process images to assistthe forensic management module 120 in extracting metadata from raw videodata. In some embodiments, the forensic analysis module 130 can identifyrelationships between objects and persons and/or can identify arelationship schema. In either embodiment, identified relationships oran identified relationship schema can be input to a relationshipbuilder.

The third party interface 140 can integrate the incident management toolinterface of the user interface 110 and other components of the system100 with the forensic management module 120. For example, the thirdparty interface can include a plurality of adapters for integratingthird parties with events that require notification.

Finally, the data repository 150 can store the data and the metadata ofraw video data, indexed metadata of various incidents, persons, andobjects, incident management data and workflow metadata, and systemstates and associated framework metrics. In some embodiments, the datarepository 150 can be only accessible via the forensic management module120 to maintain security, and in some embodiments, encapsulated data canonly be decapsulated by users with authenticated credentials. In someembodiments, the video data in the data repository 150 can be normalizedto a common form for a plurality of users, vendors, or integrators, butcan be specific to a user, vendor, or integrator so that the video datacan only be decoded with an adaptor specific to the particular user,vendor, or integrator. Furthermore, in some embodiments, databases inthe data repository 150 can use a dynamic schema that can evolve atruntime. For example, in some embodiments, runtimes can be scheduled atperiodic intervals, and data can be translated as per the dynamic schemaor rule that is adapted per the runtime requirements of a particularscheduled runtime.

In accordance with the above and below described embodiments, systemsand methods disclosed herein can passively analyze multiple video datastreams or video clips asynchronously to identify, locate, and trackobjects and to refine metadata associated with the objects for furtherforensic analysis. For example, the user interface 110 can receive userinput identifying a primary or initial person or object.

In accordance with the above and below described embodiments, systemsand methods disclosed herein can decapsulate and decode video data andtranslate raw video data into image streams, metadata, and configurationparameters that can be analyzed to identify navigation patterns of aperson or object and to identify related persons or objects. Forexample, existing video data in the data repository 150 can be convertedinto streams of optimized metadata, and the forensic analysis module 130can execute asynchronous threads of analysis over such data to searchfor and identify the primary or initial person or object, can build aninitial schema with mappings and associations to the primary or initialperson or object, can build metadata of the primary or initial person orobject based on the analysis over the data in the data repository 150,can mark possible interactions of the primary or initial person orobject with secondary persons or objects as well as associated metadata,can create a dynamic schema based on such interactions, can refineassociations between the primary or initial person or object and thesecondary persons or objects, can evolve metadata in the data repository150 with marking information for the primary or initial person orobject, the secondary persons or objections, and the interactionstherebetween, and can define persons or objects of interest andlocations or zones of interest based on the dynamic schema. In someembodiments, evolving the metadata in the data repository 150 caninclude building correlation rules that can be used by the userinterface 110 to generate incident management reports as disclosed anddescribed herein so that such reports can include all relevantinformation that has been associated with a person or object ofinterest.

In some embodiments, the defined persons and objects of interest andlocations or zones of interest can be stored in the data repository 150and be marked and plotted on a monitor of the user interface 110 or beincluded in incident management reports that can be generated on top ofthe metadata. For example, FIG. 2A and FIG. 2B are exemplary incidentmanagement reports that can be generated in accordance with disclosedembodiments. As seen, the reports can be generated based on the forensicanalysis as disclosed and described herein and can include metadata andvideo footage, an identification of persons or objects of interest andlocations or zones of interest, information related to areas, zones, orfacilities, and snapshots of events of interest.

Furthermore, in some embodiments, a generated incident management reportand the metadata used to generate the same can be translated into astoryboard that can include a set of footages and correlated events thatcan be displayed on a monitor of the user interface 150 as seen in FIG.3. For example, the storyboard shown in FIG. 3 can display persons orobjects of interest in a demultiplexed manner by rearranging the orderof video data streams from surveillance cameras to present anintelligent, for example, logical or chronological, view of the order ofevents of interest. In some embodiments, systems and methods disclosedherein can stream metadata to the user interface 150 for displaying thesame on the storyboard.

In some embodiments, systems and methods disclosed herein can generate awalk through report for a floor plan that can include the history ofpersons or objects of interest navigating the respective region as wellas occupancy details for the same, for example, the number of employees,security guards, or visitors in an identified region. In someembodiments, systems and methods disclosed herein can collect occupancydetails and the like from other sub-systems as would be known by thoseof skill in the art.

FIG. 4 is a flow diagram of a method 400 in accordance with disclosedembodiments. As seen in FIG. 4, the method 400 can include providingaccess details or credentials for video and metadata repositories in thedata repository 150 as in 405 to, for example, initialize the forensicmanagement module 120. Then, the method 400 can include periodicallypuffing video recordings and metadata information from the video andmetadata repositories as in 410 and asynchronously passing the video andmetadata information to the forensic analysis module 130 as in 415.

After the forensic module 130 receives the video and metadatainformation, the method 400 can include feeding each video recording andmetadata information to an objects classification module, an objectsinteraction classification module, and an objects relationship builderas in 420 and instructing the objects classification module, the objectsinteractions classification module, and the objects relationship builderto asynchronously analyze input data as in 425 to classify objects, toclassify object interactions, and to build object relationships. Themethod 400 can include waiting for the objects classification module,the objects interactions classification module, and the objectsrelationship builder to complete their object identification andrelationship analysis as in 430 and determining whether an operationcompletion event has been received as in 435.

When the method determines that an operation completion event has beenreceived as in 435, the method 400 can include, for each video recordingand metadata information, analyzing the results from the objectsclassification module, the objects interactions classification module,and the objects relationship builder for suspicious objects,interactions, and event sequences as in 440 and updating the analysisresults, including any object identification, interaction, anddependency graphs, as in 445 by utilizing previously stored analysisresults in a stored analysis results database 450.

The method 400 can include notifying the forensics management module 120of the analysis results as in 455, and, after the forensics managementmodule 120 receives the analysis results, the method 400 can includecontinuing to periodically pull video recordings and metadatainformation from the video and metadata repositories as in 410, passingthe analysis results to a presentation layer as in 460, and passing apresentation of the analysis results to a storyboard plug in, a userinterface renderer, or a forensic report presenter as in 465 forgeneration and presentation of the same.

It is to be understood that the systems and methods as disclosed anddescribed above, can be implemented with a transceiver device forcommunicating with a plurality of surveillance cameras or servers and amemory device for storing video data streams and metadata, each of whichcan be in communication with control circuitry, one or more programmableprocessors, and executable control software as would be understood byone of ordinary skill in the art. In some embodiments, the executablecontrol software can be stored on a transitory or non-transitorycomputer readable medium, including, but not limited to, local computermemory, RAM, optical storage media, magnetic storage media, flashmemory, and the like, and some or all of the control circuitry, theprogrammable processors, and the control software can execute andcontrol at least some of the methods described above.

Although a few embodiments have been described in detail above, othermodifications are possible. For example, the logic flows described abovedo not require the particular order described or sequential order toachieve desirable results. Other steps may be provided, steps may beeliminated from the described flows, and other components may be addedto or removed from the described systems. Other embodiments may bewithin the scope of the invention.

From the foregoing, it will be observed that numerous variations andmodifications may be effected without departing from the spirit andscope of the invention. It is to be understood that no limitation withrespect to the specific system or method described herein is intended orshould be inferred. It is, of course, intended to cover all suchmodifications as fall within the spirit and scope of the invention.

What is claimed is:
 1. A method comprising: storing a plurality of videodata streams in a data repository; storing asynchronous streams ofmetadata of each of the plurality of video data streams in the datarepository; identifying a first object captured by at least one of theplurality of video data streams; using the asynchronous streams ofmetadata to identify correlations or interactions between the firstobject and a plurality of other objects over time; and replicating astory of the first object.
 2. The method of claim 1 wherein the firstobject includes a person or a structure in a monitored region.
 3. Themethod of claim 1 further comprising using the asynchronous streams ofmetadata to classify, detect, identify, locate, or track the firstobject or to identify a relationship between the first object and theplurality of other objects.
 4. The method of claim 1 further comprisingrefining the asynchronous streams of metadata stored in the datarepository based on the correlations or the interactions between thefirst object and the plurality of other objects identified over time. 5.The method of claim 1 wherein replicating the story of the first objectincludes generating an incident management report on top of theasynchronous streams of metadata, and wherein the incident managementreport summarizes the correlations or the interactions between the firstobject and the plurality of other objects over time.
 6. The method ofclaim 1 wherein replicating the story of the first object includesgenerating a storyboard on top of the asynchronous streams of metadata,and wherein the storyboard summarizes the correlations of theinteractions between the first object and the plurality of other objectsover time.
 7. The method of claim 6 wherein the storyboard orders theplurality of video data streams to present an intelligent view of thestory of the first object irrespective of an order of the plurality ofvideo data streams in the data repository.
 8. The method of claim 6further comprising streaming the asynchronous streams of metadata to auser interface device displaying the storyboard.
 9. A method comprising:storing a plurality of video data streams in a data repository; storingmetadata of each of the plurality of video data streams in the datarepository; a classification module asynchronously analyzing theplurality of video data streams and the metadata to classify objectscaptured by the plurality of video data streams; an interaction moduleasynchronously analyzing the plurality of video data streams and themetadata to classify interactions of the objects captured; arelationship builder asynchronously analyzing the plurality of videodata streams and the metadata to build relationships of the objects;based on the objects as classified, the interactions of the objects, andthe relationships of the objects, presenting a story of at least one ofthe objects, at least one of the interactions of the objects, or atleast one of the relationships of the objects.
 10. The method of claim 9further comprising: based on the objects as classified, the interactionsof the objects, and the relationships of the objects, identifying the atleast one of the objects, the at least one of the interactions of theobjects, the at least one of the relationships of the objects, or anevent sequence of the object as suspicious; and presenting the story ofthe at least one of the objects, the at least one of the interactions ofthe objects, the at least one of the relationships of the objects, or anevent sequence of the object identified as suspicious.
 11. The method ofclaim 9 wherein the objects include a person or a structure in amonitored region.
 12. The method of claim 9 further comprising refiningthe metadata stored in the data repository based on the objects asclassified, the interactions of the objects, and the relationships ofthe objects identified over time.
 13. The method of claim 9 whereinpresenting the story of the at least one of the objects includesgenerating an incident management report on top of the metadata, andwherein the incident management report summarizes the objects asclassified, the interactions of the objects, and the relationships ofthe objects.
 14. The method of claim 9 wherein presenting the story ofthe at least one of the objects includes generating a storyboard on topof the metadata, and wherein the storyboard summarizes the incidentmanagement report summarizes the objects as classified, the interactionsof the objects, and the relationships of the objects.
 15. The method ofclaim 14 wherein the storyboard orders the plurality of video datastreams to present an intelligent view of the story of the at least oneof the objects irrespective of an order of the plurality of video datastreams in the data repository.
 16. The method of claim 14 furthercomprising streaming the metadata to a user interface device displayingthe storyboard.
 17. A method comprising: storing a plurality of videodata streams in a data repository; storing metadata of each of theplurality of video data streams in the data repository; a classificationmodule asynchronously analyzing the metadata to classify objectscaptured by the plurality of video data streams; an interaction moduleasynchronously the metadata to classify interactions of the objectscaptured; a relationship builder asynchronously analyzing the metadatato build relationships of the objects; based on the objects asclassified, the interactions of the objects, and the relationships ofthe objects, presenting a story of at least one of the objects, at leastone of the interactions of the objects, or at least one of therelationships of the objects.
 18. The method of claim 17 furthercomprising refining the metadata stored in the data repository based onthe objects as classified, the interactions of the objects, and therelationships of the objects identified over time.
 19. The method ofclaim 17 wherein presenting the story of the at least one of the objectsincludes generating an incident management report on top of themetadata, and wherein the incident management report summarizes theobjects as classified, the interactions of the objects, and therelationships of the objects.
 20. The method of claim 17 whereinpresenting the story of the at least one of the objects includesgenerating a storyboard on top of the metadata, and wherein thestoryboard summarizes the incident management report summarizes theobjects as classified, the interactions of the objects, and therelationships of the objects.